I. Technical Field
The present invention relates generally to computer processing and more specifically, to a system and method for evolving appropriate connections in sparse neural networks.
II. Description of the Related Art
A neural network is a system composed of many simple processing elements operating in parallel which has a natural propensity for storing experiential knowledge and making it available for use. It resembles the brain in two respects: 1) knowledge is acquired by the network through a learning process, and 2) interneuron connection strengths known as synaptic weights are used to store the knowledge.
Neural network devices (NNDs) are used to process information using neuron-like units connected by variable strength “synapses”. These devices can be realized in either hardware or software, and they perform massively parallel computations that are useful in pattern recognition, finance and a host of other applications. The computations performed by the devices are determined by the strengths of the connections, which constitute the computational “program”. There are various procedures for setting the strengths of the connections, mostly variants of the so-called “Hebbian” rule, a type of feedforward neural network in which the connection strengths are determined in a training phase by the past history of activity of both input and output neurons. The most well known variant is “backpropagation”. The backpropagation algorithm compares an output (result) that was obtained with the result that was expected. It then uses this information to systematically modify the weights throughout the neural network to better predict future outputs.
Typically, existing neural network systems have an architecture that is fully connected; that is, all its computing units are in contact (fully connected) with each other. This necessitates the examination of activity across each connection, which, due to the potentially vast number of connections in fully connected architectures, is a formidable task. Thus, current neural network technologies (either in hardware or software) suffer from overfitting and often require excessive computational resources or expert architectural specification. Although some existing neural networks do use limited connectivity, they usually involve a fixed architecture, in which there are a small number of connections that do not change and thus lack adaptive ability. In addition, to limit the number of connections, some previous approaches have either involved simply pruning away weak connections, or using separate evolutionary algorithms to find suitably sparsely-connected NNDs that can then be programmed by conventional activity-dependent methods.
Accordingly, an efficient and accurate technique for training neural networks which both simplifies the task of setting connection strengths and allows new connections to be created as training proceeds, is highly desirable.